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The  Test  &  Evaluation  (TScE)  commu¬ 
nity  likes  to  talk  about  realistic  opera¬ 
tional  testing.  Today,  weapons  systems 
are  being  rushed  overseas  to  face  the 
real  test  in  real  —  not  realistic  — 
operations.  Our  nation  is  at  war  and  T&E  must  do  all 
it  can  in  support.  Rapid  acquisition  is  a  worthy 
response,  and  finding  failure  modes  early  is  an  obvious 
way  to  avoid  delay  when  problems  are  discovered, 
avoid  costly  rework  for  the  fix,  or  preclude  fielding 
systems  with  problems.  There  are  examples: 

•  The  Stryker  Mobile  Gun  System’s  coaxially 
mounted  7.62-mm  machine 
gun.  In  predeployment  testing, 
the  system  appeared  to  function 
satisfactorily.  Yet,  when  handed 
off  to  troops  and  fired  in  new 
equipment  training,  the  machine 
gun’s  bore-sight  failed. 

•  Two  air  weapons  systems:  Small 
Diameter  Bomb,  and  Joint  Air- 
to-Surface  Standoff  Missile. 

Both  deployed  systems  were 
grounded.  The  weapons  systems 
were  under-designed,  not  tested 
for  realistic  operational  tempo, 
and  experienced  unacceptable 
reliability  and  related  operation¬ 
al  performance  failures. 

•  The  MH-60S  Armed  Helicopter  Weapons 
System.  HeUfire  missiles  hanging  from  external 
launchers  —  forward  of  and  level  with  the  open 
cabin  doors  —  created  a  hazard  for  the  helicop¬ 
ter’s  GAU  21  gunner.  In  a  post  OT  training 
incident,  blast  from  a  Hellfire  broke  pins  securing 
the  gun  and  forced  the  gun  barrel  around,  causing 
it  to  pierce  the  fuselage.  If  the  gunner  had  been  in 
contact  with  the  weapon,  results  could  have  been 
catastrophic. 

In  the  above  examples,  more  realistic  test  environments 
and  operations  would  have  enabled  discovery  of  the 
problems.  The  need  to  find  problems  earlier  is 
recognized  inside  and  outside  the  Department  of 
Defense  (DoD).  DoD  promulgated  T&E  policy 
changes  in  a  December  22,  2007,  memorandum 


(reprinted  in  the  appendix  following  this  article  and 
available  at  https://akss.dau.mil/Documents/Policy/ 
TE-Policy-Memo-Dec-2007.pdf  ).  The  goal  of  this 
policy  is  “early  identification  of  technical,  operational, 
and  system  deficiencies,  so  that  appropriate  and  timely 
corrective  actions  can  be  developed  prior  to  fielding  the 
system.”  The  leverage  of  finding  problems  early  in 
system  design  is  obvious,  but  unfortunately  we  often 
discover  problems  late,  even  after  production  commit¬ 
ments  have  been  made.  In  this  issue  Darlene  Mosser- 
Kerner  explains  the  new  policy  and  its  impact  on 
revitalizing  T&E. 

The  need  to  find  problems  earlier 
is  recognized  outside  the  Depart¬ 
ment  of  Defense.  The  basis  for  the 
new  T&E  policy  is  a  DoD  Report 
which  cites  recommendations  by  the 
Government  Accountability  Office, 
Defense  Acquisition  Performance 
Assessment,  and  National  Research 
Council.  The  report  recommended 
that  DoD:  lessen  dependence  on 
testing  late  in  development;  consol¬ 
idate  Developmental  Testing  (DT) 
and  Operational  Testing  (OT);  and, 
require  DT  to  have  an  operational 
perspective.  In  all  instances,  the 
implied  goal  is  early  identification 
of  operational  failure  modes  and  system  deficiencies. 

At  the  same  time,  Dr.  Charles  McQueary,  Director 
for  Operational  Test  and  Evaluation  (DOT&E), 
established  a  priority:  “Enhance  operational  realism 
in  early  tests,  including  DT.”  ITEA  should  be  praised 
for  selecting  this  theme  for  The  ITEA  Journal. 

The  DOT&E  desired  end-state  is,  “Sufficient 
operational  insights  gained  prior  to  design  reviews  and 
acquisition  decision  points  to  influence  system  design 
and  reduce  surprises  in  operational  test.”  As  Dr. 
McQueary  stated  in  his  guest  editorial  in  the  September 
2007,  ITEA  Journal:  “OT&E  should  be  a  time  of 
confirmation,  not  discovery.”  The  focus  of  employing 
operational  realism  in  early  tests  is  on  designing  the 
system  to  operate  effectively  in  the  environment  (with 
threat  conditions)  and  with  the  system  operators  and 
maintainers  anticipated  when  the  system  is  deployed. 
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Operational  realism  may  be  easiest  to  imagine  when 
thinking  about  system  level  tests  —  when  the  system  or 
a  prototype  is  available.  But  operational  realism  has  a 
role  very  much  earlier  during  design,  when  components, 
subsystems,  and  operational  procedures  are  chosen. 

What  does  it  mean  to  include  operational  realism  in 
testing  early?  Enhanced  operational  realism  should  be 
part  of  demonstrating  technology  readiness.  Often, 
systems  enter  design  and  development  at  Milestone  B 
(MS  B)  with  insufficient  technology  maturity.  Public 
law  (PL  109-163,  Section  801)  now  requires  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology  and 
Logistics  (USD(AT&L))  to  certify  technology  has  been 
demonstrated  in  a  relevant  environment.  This  certifica¬ 
tion  could  add  discipline  to  acquisition  guidance  calling 
for  demonstrating  Technology  Readiness  Level  6  prior 
to  MS  B,  i.e.,  demonstrating  the  technology  in  an 
operationally  relevant  environment  before  MS  B. 
Testers  should  be  involved  in  the  technology  demon¬ 
stration,  and  influence  the  characterization  of  the 
“relevant  environment”  so  that  it  is  both  operationally 
relevant  and  consistent  with  subsequent  testing. 

Operational  realism  in  early  testing  should  mean 
that  components  are  tested  in  a  relevant  environment. 
Lor  example,  the  vibrational  environment  on  the  center 
pylon  of  the  L-15  necessitated  major  redesign  of  the 
original  AMRAAM  missile  because  no  one  character¬ 
ized  that  acoustic  environment  and  tested  the  compo¬ 
nents  and  system  in  those  conditions.  Typically, 
designers  assume  the  systems  with  which  they  interface 
are  characterized  by  the  design  specifications  of  the 
item  in  development.  Often  the  truth  in  the  field  is 
different.  Lor  example,  electrical  power  is  often 
“dirtier”  (noisier)  than  specified.  Testing  system 
components  in  a  realistic  environment  —  under 
realistic  stress  —  can  save  substantial  system  develop¬ 
ment  time.  Dr.  Cliff  Duncan,  the  second  Director  for 
OT&E,  once  told  me  that  with  avionics  he  found  the 
first  places  to  look  for  trouble  were  in  the  behavior  of 
the  power  supply  and  the  connector.  Shouldn’t  that  be 
the  first  place  we  test  components  too? 

Enhanced  operational  realism  should  be  part  of 
determining  whether  commercial-off-the-shelf  (COTS) 
technology,  software,  or  system  components  will  actually 
perform  as  needed  when  in  real  operations.  COTS 
functionahty  can  be  highly  desirable  but  a  key  question  is: 
does  COTS  performance  hold  up  in  operations? 
Obviously,  the  military  environment  can  be  more 
demanding.  System  designers  should  step  up  to  the 
challenge  of  assessing  COTS  in  the  operational  environ¬ 
ment.  Testers  are  obligated  to  identify  COTS  risks  by 
testing  it,  early,  in  the  proper  environment. 


Enhanced  operational  realism  means  we  need  to 
employ  real  operators  to  operate  the  system.  We  have 
had  cases  where  equipment  that  was  effective  when 
used  by  other  nations,  was  not  so  effective  when  our 
forces  used  it.  Part  of  the  reason  was  that  another 
country  could  man  the  system  with  highly  educated 
soldiers  who  understood  much  of  the  basic  chemistry 
that  impacts  system  operation.  You  may  have  heard 
stories  of  systems  failing  because  the  operator  used 
them  inappropriately.  The  first  operational  test  of  the 
SINCGARS  radio  had  to  be  stopped.  Soldiers  when 
ordered  to  move  the  radio  grabbed  it  by  the  antenna  — 
which  broke  —  leading  to  a  mission  failure.  The  radio 
did  not  have  a  handle.  The  point  is,  include  those 
operators  early  so  we  will  understand  differences 
between  the  systems  operated  by  system  developers 
and  warfighter  operators. 

Enhanced  operational  realism  also  means  that  all  the 
interactions  and  interfaces  that  have  to  work  for  a 
mission  to  be  successful  are  checked  before  the  design 
is  finalized.  More  general  than  connectors  and  power 
supplies,  the  age  of  net-centric  operations  and  service 
oriented  architectures  requires  sharing  data  and 
coordinating  activity  of  separately  developed  services. 
If  “n”  services  must  work  for  mission  success,  and  each 
service  has  a  probability  of  success  “x,”  then  —  to  a  first 
approximation  —  the  mission  will  have  a  probability  of 
success  of  only  x".  Lor  example,  if  you  have  six  services 
that  have  to  work,  and  you  only  want  0.8  chance  of 
success,  then  each  service  has  to  exceed  0.96  probability 
of  success!  That  does  not  consider  the  case  when  the 
services  are  correlated,  but  they  will  be.  Service 
oriented  architectures  will  require  more  unit  level 
testing,  much  more  regression  testing,  and  lots  of  end- 
to-end  testing.  It  win  be  essential  in  testing  a  new 
service  that  simulated  inputs  from  other  services  be 
realistic. 

So,  it  is  important  for  testers  to  enhance  operational 
realism  in  T&E  in  every  way  they  can  imagine. 
Operational  realism  in  early  T8cE  can  improve  the 
chances  of  success  in  rapid  fielding  and  in  OT.  It  is  a 
way  we  can  better  support  our  deployed  forces.  □ 

Dr.  Ernest  Seglie  is  science  advisor.  Office  of  the 
DOT^E,  the  Pentagon,  Washington,  D.  C.  He  provides 
scientific  and  technical  guidance  on  the  overall  approach  to 
DoD  evaluation  of  the  operational  effectiveness  and 
suitability  of  major  DoD  weapon  systems;  provides 
technical  review  of  test  reports;  and  serves  as  chief  technical 
advisor  to  the  DOT&'E. 
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OFFICE  OF  THE  SECRETARY  OF  DEFENSE 
WASHINGTION,  DC  20301-1000 


DEC  2.2  2007 


MEMORANDUM  FOR:  SEE  DISTRIBUTION 
SUBJECT:  Test  and  Evaluation  Policy  Revisions 

The  fundamental  purpose  of  test  and  evaluation  is  to  provide  knowledge  to  assist 
in  managing  the  risks  involved  in  developing,  producing,  operating,  and  sustaining 
systems  and  capabilities. 

T&E  measures  progress  in  both  system  and  capability  development.  T&E 
provides  knowledge  of  system  capabilities  and  limitations  to  the  acquisition  community 
for  use  in  improving  the  system  performance,  and  the  user  community  for  optimizing 
system  use  in  operations.  T&E  expertise  must  be  brought  to  bear  at  the  beginning  of  the 
system  life  cycle  to  provide  earlier  learning  about  the  strengths  and  weaknesses  of  the 
system  under  development.  The  goal  is  early  identification  of  technical,  operational,  and 
system  deficiencies,  so  that  appropriate  and  timely  corrective  actions  can  be  developed 
prior  to  fielding  the  system.  Consequently,  to  achieve  this  goal  we  have  decided  to 
immediately  implement  the  following  policies: 

•  Developmental  and  operational  test  activities  shall  be  integrated  and  seamless 
throughout  the  system  life  cycle.  As  technology,  software,  and  threats  change, 
follow-on  T&E  should  be  used  to  assess  current  mission  performance  and  inform 
operational  users’  during  the  development  of  new  capability  requirements. 

•  Evaluations  shall  include  a  comparison  with  current  mission  capabilities  using 
existing  data,  so  that  measurable  improvements  can  be  determined.  If  such  evaluation 
is  considered  cost  prohibitive  the  Service  Component  shall  propose  an  alternative 
evaluation  strategy. 

•  T&E  should  assess  improvements  to  mission  capability  and  operational  support  based 
on  user  needs  and  should  be  reported  in  terms  of  operational  significance  to  the  user. 
Consequently,  evaluations  shall  be  conducted  in  the  mission  context  expected  at  time 
of  fielding,  as  described  in  the  user’s  capability  document,  and  consider  any  new 
validated  threat  environments  that  will  alter  operational  effectiveness. 

•  To  maximize  the  efficiency  of  the  T&E  process  and  more  effectively  integrate 
developmental  and  operational  T&E,  evaluations  shall  take  into  account  all  available 
and  relevant  data  and  information  from  contractor  and  government  sources. 
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•  Operational  evalmtora  will  continue  to  fulfill  their  statutory  roles  in  providing 
assessments  of  operational  effectiveness,  operational  suitability,  and  survivability  to 
the  MDA.  In  addition,  program  managers  shall  report  the  results  of  completed 
developmental  testing  to  the  milestone  decision  authority  at  milestones  B  and  C.  The 
report  shall  identify  strengths  ^d  weaknesses  in  meeting  the  warfighters’  documented 
needs  based  on  developmental  evaluations.  The  operational  evaluators  assessment 
will  be  provided  to  the  MDA  at  the  full  rate  production  review. 

•  To  realize  the  benefite  of  modeling  and  simulation,  T&E  will  be  conducted  in  a 
continuum  of  live,  virtual,  and  constructive  system  and  operational  environments. 


These  policies  will  be  incoiporat^i  in  die  next  revision  to  DoDI  5000.2. 


Dr.  Charles  E.  McQueary 
Director,  Operational  Te^  &  Evaluisii 
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